An optical instrument is generally required to produce a geometrically consistent image of a given object, where each point is imaged as a point in the image. The image is generally formed in accordance with some predefined imaging model, which in this case, is assumed to be a projective camera. The departure of practical optical systems from ideal behavior leads to the introduction of aberrations in the resulting images. The aberration here is lens distortion, a well known monochromatic aberration, which is present in both color and greyscale imaging devices. Its nature is predominantly a radial geometric displacement of pixels giving a barrel or a pincushion effect but without loss of image quality.
Lens distortion is a thorn in the side of many relatively straightforward image analysis tasks. It comprehensively degrades the accuracy of measurements made in real images, where pixels in a normal perspective camera can typically be displaced by up to ten percent of the image width.
One known method for addressing lens distortion is in conjunction with a complete calibration of the camera system. Complete camera calibration essentially means that the camera's internal and external parameters are obtained. Internal parameters roughly relate to the camera lens, such as focal length and optical axis. Lens distortion parameters may be considered as an add-on to the set of internal camera parameters. The external parameters relate to the positioning of the camera. Such conventional camera calibration methods are disclosed in many research papers. The importance of distortion in this regard has seen the utter dominance of these more complicated algorithms at the expense of those that do not consider distortion.
Despite this, many applications do not require the full complement of internal camera parameters and the relative orientation in relation to some calibration target. Non-metric applications differ in that they do not necessarily employ the lens or external characteristics in determining distortion.
In the 1960's Duane C. Brown, working in the area of photogrammetry, proposed a method for determining lens distortion based on the truism that straight lines must be imaged as straight lines. This technique, published in (Brown, 1971), and with extensions in (Fryer and Brown, 1986), became known as the ‘plumb line’ method, where initially fine white thread was stretched by plumb bobs which were stabilized in an oil bath. A comprehensive historical review is given in Clarke and Fryer (1998). This technique was adopted by the machine vision community where simplified versions of the plumb line method were presented, e.g. Prescott and McLean (1997b), (also as patent Prescott and McLean (1997a) XP000683415) Haneishi et al. (1995a) (also as patent Haneishi et al. (1995b) XP000527216) Poulo and Gorman (1999) U.S. Pat. No. 6,002,525 and Asari et al. (1999) all describe a similar truism based correction for the correction of distortion using images of co-linear points. Similarly, high level distortion is considered in Brauer-Burchardt and Voss (2001), however the real example shown, with a quoted residual distortion of ±0.7 pixels, resembles a quite benign distortion level. Since these methods only estimate distortion, they are sometimes referred to as non-metric calibration.
An intrinsic problem for the multiple line based methods is that it becomes intractable to form geometric relationships between a distorted line segment and its true projection. An alternating approach was employed, for example in Devernay and Faugeras (2001), Tzu-Hung (2003) and Bing (1999) EP0895189, which iteratively adjusts the distortion parameters in order to minimize the line fitting to the distorted line segments. No meaningful geometric relationship exists between this objective error and the distortion parameters, hence no analytical derivatives are available. This results in slow convergence and can become unstable for elevated distortion levels, unless special steps are taken, as in Swaminathan and Nayar (2000). In this non-metric approach Swaminathan and Nayar (2000) reformulated the objective function in distorted space instead of the usual undistorted space. This is done by performing a further search at each alternation to find the location of a point closest to the considered distorted point, but that lies exactly on the line fitted to the current undistorted point estimates. The reported results showed improved robustness to noise for simulated data but no meaningful performance was reported in the real case.
Another approach has been suggested in Ahmed and Farag (2001) where the curvature of detected lines are used to estimate the parameters of the derivative distortion equation. However, the simulation results showed abysmal performance in the presence of noise, while the real results lacked a qualitative evaluation.
A more standard means of calibrating distortion is with the simultaneous estimation of a camera's extrinsic and intrinsic parameters. Tsai's method (Tsai, 1987) involves simultaneously estimating, via an iterative numerical optimization scheme, the single distortion parameter and some internal parameters such as focal length, given the 3D position of a set of control points. The external parameters or position of the camera is already computed in a previous step. The disadvantage of this approach is that it requires known 3D control points and in return offers relatively low accuracy for all but simple distortion profiles. Algorithmic variations on this principal have been proposed by some, such as Weng et al. (1992) and Wei and Ma (1994) using more appropriate models for lens distortion. These methods also require known 3D control points.
The generation of distortion corrected images is investigated in Heikkila and Silven (1997), while Heikkila (2000) describes a similar technique that requires 3D control points or multiple image sets of 2D control points. An alternative method also based on multiple sets of 2D control points has been advanced in Zhang (1998, 2000) and Sturm and Maybank (1999). This technique addresses distortion through an alternating linear least squares solution which is then iteratively adjusted in a numerical minimization including all estimation parameters. The relative complexity of these techniques is increased by the inclusion of lens distortion.
On the other hand there are many situations where only distortion removal is required, not the full complement of intrinsic and extrinsic parameters. An example is in the estimation of multiple view geometry in real images, where techniques have been specifically developed to accommodate lens distortion. Zhang (1996) investigates the possibility of simultaneously estimating distortion parameters and the Fundamental Matrix. The results conclude that this is possible if noise is low and distortion is high. Fitzgibbon (2001) (with patent Fitzgibbon (2003) GB2380887), Micusik and Pajdla (2003) and Barreto and Daniilidis (2004) use an alternative model for distortion, leading to a polynomial Eigenvalue problem and a more reliable estimation of distortion and geometry. Stein (1997) took the reverse approach and used the error in Fundamental Matrix estimation as an objective error to estimate distortion parameters.
Alternative methods of distortion calibration exist, where control points correspondences are abandoned in favor of distortion free scenes. These scenes are then imaged by the camera system, whereupon an image alignment process is conducted to correct for distortion. Lucchese and Mitra (2003) describe a technique where the distorted image is warped until it registers (in intensity terms) with the reference image. A similar technique using a coarse filter to find registration is described in Tamaki (2002) (with patent Tamaki et al. (2002) US2002057345) while Sawhney and Kumar (1999) (also with patent Kumar et al. (1998) WO9821690) describe a registration technique that does not require an undistorted reference image. Instead, multiple images are registered for the generation of a mosaic image, for example such as a panoramic resulting from the combination of several different views, and distortion is simultaneously estimated. These techniques have a very high computational overhead, with twenty minutes quoted in Tamaki (2002).
A final class of non-metric calibration methods are based on distortion induced high-order correlations in the frequency domain. Farid and Popescu (2001) describe a technique, however its performance is poor in comparison with regular camera calibration techniques and it also appears to be slightly dependent on the image content. Yu (2004) further develops this approach with alternative distortion models and reports accuracy approaching that achieved with regular camera calibration if the source image is of a regular calibration target. Finally, a means of fabricating a curved CCD array has been suggested by Gary (2003) US2003141433. In the lens distortion profile is copied in the array by a series of line segments, thus the resulting image appears distortion free.